Value-based presentation of user-selectable computing actions

ABSTRACT

Computer-implemented methods for proposing actions to a user to select based on the value of the actions to the user are provided. In one aspect, a method includes receiving content selected by a user from a device, and identifying a plurality of actions to be executed that are associated with a referent entity identified from the selected content, wherein each action has an associated value to the user. The method also includes providing, for display, an identifier of each of at least a subset of the actions based on their associated values. Systems and machine-readable storage media are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit of priority under 35 U.S.C.§119 from U.S. Provisional Patent Application Ser. No. 61/543,752entitled “Referent Determination From Selected Content,” filed on Oct.5, 2011, and U.S. Provisional Patent Application Ser. No. 61/641,242entitled “Value-Based Presentation of User-Selectable ComputingActions,” filed on May 1, 2012, the disclosures of which are herebyincorporated by reference in their entirety for all purposes.

BACKGROUND

1. Field

The present disclosure generally relates to the transmission of dataover a network, and more particularly to the use of a computing deviceto identify data communicated over a network.

2. Description of the Related Art

Users often want to perform certain actions using content, such as textand images, displayed on their devices. For example, if a user isviewing a web page describing a restaurant, the user may want to createa reservation at the restaurant or obtain directions to the restaurantfrom the user's current location. Facilitating the user's purpose (e.g.,to create a reservation or obtain directions) usually requires the userto perform several input actions with the device. For example, for arestaurant, a user usually finds the name of the restaurant displayed onscreen, carefully selects the name, copies the selected content, pastesthe select content into a search query, submits the search query,selects the restaurant's web site from the search results, determines ifthe user can make a reservation on the web site, and then providesinformation for the reservation. With touchscreen devices, such asdesktop or mobile devices with touch input, facilitating the user'spurpose can be especially cumbersome because the user is often limitedto touch input.

SUMMARY

According to one embodiment of the present disclosure, acomputer-implemented method for proposing actions to a user to selectbased on the value of the actions to the user is provided. The methodincludes receiving content selected by a user from a device, andidentifying a plurality of actions to be executed that are associatedwith a referent entity identified from the selected content, whereineach action has an associated value to the user. The method alsoincludes providing, for display, an identifier of each of at least asubset of the actions based on their associated values.

According to another embodiment of the present disclosure, a system forproposing actions to a user to select based on the value of the actionsto the user is provided. The system includes a memory that includesuser-selectable content, and a processor. The processor is configured toreceive content selected by a user from a device, and identify aplurality of actions to be executed that are associated with a referententity identified from the selected content, wherein each action has anassociated value to the user. The processor is also configured toprovide, for display, an identifier of each of at least a subset of theactions based on their associated values. The associated value for anaction is a function of an estimated average amount of time it takes auser to manually perform the action.

According to a further embodiment of the present disclosure, amachine-readable storage medium includes machine-readable instructionsfor causing a processor to execute a method for proposing actions to auser to select based on the value of the actions to the user isprovided. The method includes receiving content selected by a user froma device, and identifying a plurality of actions to be executed that areassociated with a referent entity identified from the selected content,wherein each action has an associated value to the user that is afunction of an estimated average amount of time it takes a user tomanually perform the action. The method also includes providing, fordisplay, in a list format at least one of a text or image thatidentifies each of at least a subset of the actions. The text or imagefor each action in the list is displayed in decreasing order of theassociated value for the action.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide furtherunderstanding and are incorporated in and constitute a part of thisspecification, illustrate disclosed embodiments and together with thedescription serve to explain the principles of the disclosedembodiments. In the drawings:

FIG. 1 illustrates an example architecture for identifying an entitybased on selected content.

FIG. 2 is a block diagram illustrating an example client and server fromthe architecture of FIG. 1 according to certain aspects of thedisclosure.

FIG. 3 illustrates an example process for identifying an entity based onselected content.

FIGS. 4A-4E are example illustrations associated with the exampleprocess of FIG. 3.

FIG. 5 is a block diagram illustrating an example computer system withwhich the client and server of FIG. 2 can be implemented.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth to provide a full understanding of the present disclosure. It willbe apparent, however, to one ordinarily skilled in the art that theembodiments of the present disclosure may be practiced without some ofthese specific details. In other instances, well-known structures andtechniques have not been shown in detail so as not to obscure thedisclosure.

The disclosed system decides how to present actions, associated with apredicted object (or “referent”) of a user's selected content, based onthe value of the actions to the user. In certain aspects, actions arepresented for display that are less commonly selected by users but wouldsave a user significantly more time than other actions that are morecommonly selected by users. For example, a user selects the text “welove Sushi ABC out of all of the places we've been to.” The disclosedsystem determines that a referent from the selected text is therestaurant Sushi ABC, and then predicts various user purposes associatedwith the referent, such as that the user wants to (1) obtain directionsto Sushi ABC, (2) make a reservation at Sushi ABC, or (3) read reviewsabout Sushi ABC. The system further determines that 85% of past usershave selected option (1), which would take a user about 30 seconds tomanually complete, 5% of past users have selected option (2), whichwould take a user about 5 minutes to manually complete, and 10% of pastusers have selected option (3), which would take a user about 30 secondsto manually complete. Although option (2) has a relatively lowlikelihood of selection, option (2) is presented along with option (1)(but option (3) is not presented) because option (2) is deemed to be ahigh value option because of how long it would take a user to manuallycomplete the option were it not presented.

Although many examples provided herein describe a user's information(e.g., user selections and other actions) being stored in memory, theuser can, at any time, delete the user information from memory and/oropt out of having the user information stored in memory. Additionally,the user can, at any time, adjust appropriate privacy settings toselectively limit the types of user information stored in memory, orselect the memory in which the user information is stored (e.g., locallyon the user's device as opposed to remotely a server). The userinformation does not include and/or share the specific identification ofthe user (e.g., the user's name) unless otherwise specifically providedor directed by the user.

FIG. 1 illustrates an example architecture 100 for identifying an entitybased on selected content. The architecture 100 includes servers 130 andclients 110 connected over a network 150.

One of the many servers 130 is configured to host an entity database.For purposes of load balancing, multiple servers 130 can host the entitydatabase (or portions thereof). As discussed herein, the entity databasereceives a copy of content selected by a user of one of the clients 110,and then identifies at least one entity from the entity database that isbeing referred to in the selected content. The entity being referred toin the selected content (i.e., the “referent entity” or “referent”) canbe identified, for example, by identifying one or many entitiesappearing in the selected content or inferred from the selected content(e.g., using a context in which the content is selected), and comparingthe identified entities or other related entities to entities in theentity database, along with a context in which the content was selected.The context can be based on, for example, a current or past location ofthe client 110, a previous use of the selected content by the user oranother user, a search query, information on another user associatedwith the user, the file or source from which the selected content wasselected, and the remaining content from which the selected content wasselected (e.g., based on text, audio, or image data surrounding theselected content). An identifier of the referent entity can then beprovided to the client 110. After the entity is selected on the client110 by the user, at least one action associated with the entity to beexecuted can be identified based on a context in which the entity isselected on the client. Information on the action can then be providedto the client 110 for display and selection by a user.

The servers 130 can be any device having an appropriate processor,memory, and communications capability for hosting the entity database.The clients 110 to which the servers 130 are connected over the network150 can be, for example, desktop computers, mobile computers, tabletcomputers (e.g., including e-book readers), mobile devices (e.g., asmartphones or personal digital assistants), set top boxes (e.g., for atelevision), video game consoles, or any other devices havingappropriate processor, memory, and communications capabilities forinteracting with content. The network 150 can include, for example, anyone or more of a personal area network (PAN), a local area network(LAN), a campus area network (CAN), a metropolitan area network (MAN), awide area network (WAN), a broadband network (BBN), the Internet, andthe like. Further, the network 150 can include, but is not limited to,any one or more of the following network topologies, including a busnetwork, a star network, a ring network, a mesh network, a star-busnetwork, tree or hierarchical network, and the like.

FIG. 2 is a block diagram 200 illustrating an example server 130 andclient 110 in the architecture 100 of FIG. 1 according to certainaspects of the disclosure. The client 110 and the server 130 areconnected over the network 150 via respective communications modules 218and 238. The communications modules 218 and 238 are configured tointerface with the network 150 to send and receive information, such asdata, requests, responses, and commands to other devices on the network.The communications modules 218 and 238 can be, for example, modems orEthernet cards.

The server 130 includes a processor 236, a communications module 238,and a memory 232 that includes an entity database 234. In certainaspects, the entity database 234 includes a listing of entities, each ofwhich could be a possible referent entity of selected content. Theentity database 234 can be stored in any format well known to one ofordinary skill in the art, such as, but not limited to, an array, alinked list, a hash-table, a heap, a tree, a stack, a graph, or a queue.The entities include people, places, and things. For example, an entitycan be a person, a product being sold, or a business such as arestaurant. The entity of the selected content “this literature is alarge body of literature in the English and American languages producedby the Hoysala Empire (1025-1343) in what is now southern India,” can,for example, be identified as Hoysala literature.

Entities in the entity database 234 can be tagged or otherwiseidentified with certain descriptors (e.g., attributes or properties).The descriptors can be pre-defined by an administrator of the server 130or created by other users. For example, an entity such as a restaurantthat is a place (e.g., a café at a specific location in New York City)can have associated descriptors “location,” “restaurant,” and “phonenumber,” and an entity that is a person such as George Washington canhave an associated descriptor “notable person” (e.g., the entity GeorgeWashington). The person can be notable for many reasons, such as, inthis case, being the first president of the United States. Exampledescriptors include, for example, restaurant, store, hotel, automatedteller machine, airport, place of worship, notable place, notableperson, administrator, product, movie, museum, and software. Descriptorscan include any descriptive label for an entity, and are not limited tothe example descriptors disclosed herein.

An entity listed in the entity database 234 can be associated with oneor many user purposes and/or actions based on an associated descriptor.Specifically, each entity can be associated with one or many purposes,and each of the one or many purposes for the entity can be associatedwith one or many actions. As discussed herein, a “purpose,” “userpurpose,” or “predicted purpose” is what a user wants to do or find outwith respect to an entity that is deemed to be the referent of selectedcontent. An “action” or “user action” is a menu item that is provided toa user on a client 110 that facilitates the user accomplishing apurpose. A collection of purposes can be called a “task.” Examplepurposes include “play” (e.g. for games and sports), “rate” or“evaluate,” “travel to,” “contact,” “communicate,” “share,” “record,”“remember,” dine,” “consume,” “experience” or “enjoy” (e.g. art, music),“reserve” (tickets, etc.), “compare,” “learn,” “study,” “understand,”“purchase,” “repair,” “fix,” “teach,” “cook,” and “make.” For theexample purpose “dine,” an example sub-purpose can be “eat dinner,” fromwhich example sub-purposes can be “make reservation,” “get directions,”and “find parking.”

For example, for an entity with a descriptor “restaurant,” a purposemight be to get to the restaurant and a corresponding action might be toprovide the user with directions from the user's current location on theuser's mobile client 110 to an address associated with the restaurant.Another corresponding action might be to request a taxi to pick up theuser from the user's current location and deliver the user to therestaurant. Other example purposes for selecting an entity with adescriptor of “restaurant” include calling, making a reservation,reading reviews, and saving contact information. Similarly, if theuser's purpose is to eat dinner, then it can include the purposes ofchoosing a restaurant, making a reservation, and traveling to therestaurant. As another example, for an entity with a descriptor“software,” a purpose might be to install the software and an actionmight be to provide the user with a one click option to install thesoftware on the user's desktop client 110. In certain aspects, actionsare associated with corresponding purposes.

In certain aspects, an entity can be associated with a purpose (andcorresponding action(s)) that is valid when a group of similar entitiesis selected. For example, if selected content includes three entitieswith a descriptor “restaurant,” and all three of the entities aretogether considered the referent of the selected content, then the userpurpose can be to compare the restaurants. The associated action can beto provide the user with a comparison of reviews or proximities of thethree restaurants.

Entities, purposes, and actions can be added to the entity database 234manually or automatically. In response, the entity database 234 can beupdated periodically or in real-time. Entities can be added to theentity database 234 manually, for example, by a user adding or removinga listing to the entity database 234 for a new local restaurant thatopened near the user or old restaurant that closed near the user. Asanother example, if an entity in the entity database 234 is notassociated with a telephone number, a user (e.g., owner of the entity)can provide a telephone number for the entity using an appropriateinterface for the entity database 234. An entity's listing can include,for example, a name identifier of the entity, a web site associated withthe entity, a history of the entity, contact information for the entity,relationships the entity has, images associated with the entity,availability of the entity, a price or value associated with the entity,and a location of the entity. For example, for a restaurant, the listingcan include the restaurant's name, location, store hours, menu, history,and relevant descriptors such as “place” and “restaurant.” Actions canbe added to the entity database 234 manually, for example, by a useradding or removing an action from a list of actions associated with anentity in the entity database 234. This can occur, for example, by auser providing instructions on how to perform a new action in the list(e.g., by providing step by step instructions on how to perform the newaction) or by providing input parameters to an action template (e.g., byproviding address information for a directions template for a locationentity).

Entities, purposes, and actions can be added to the entity database 234automatically, including additions by data sources/feeds, inferentialadditions, and programmatic additions. Data source automatic additionsinclude, for example, the processor 212 periodically crawling Internetresources (e.g., white pages, yellow pages, encyclopedias, socialnetworks, mapping databases, online stores, online reviews, other onlineindex/data providers, etc.) that list or include information onentities, and then import that information on the entities to the entitydatabase 234. Entities, purposes, and actions can also be added orremoved to the entity database 234 inferentially, for example, inresponse to actions taken by (or not taken by) users, such as searchterms, web pages, blogs, emails, and/or comments commonly entered byusers or proposed entities frequently ignored by users. For instance, auser searches for a specific restaurant, the specific restaurant isidentified as the entity being referred to, and a telephone number forthe restaurant is not in the entity database 234. If the user thereafteruses the client 110 to make a telephone call within a certain amount oftime (e.g., within a minute after searching for the specificrestaurant), then the telephone number dialed can be added to the entitydatabase 234 as the telephone number for the restaurant. As anotherexample, a user searches for a specific restaurant, the specificrestaurant is identified as the entity being referred to, and an addressfor the restaurant is not in the entity database 234. If the userthereafter changes geographical position and indicates the user is atthe restaurant, the geographic location of the user (e.g., using aGlobal Positioning System (GPS) sensor in the client 110) can be addedto the entity database 234 as the location of the restaurant. As afurther example, the entity database 234 includes an entry for abusiness called “Chez Paul” that includes a telephone number for ChezPaul. The entry does not, however, indicate that Chez Paul is arestaurant. If there is an indication that a user is looking for arestaurant (e.g., by inference or because the user explicitly providesthe indication), then when the user chooses to call Chez Paul from theclient 110, the entity database 234 can update the entry for Chez Paulto indicate it is a restaurant (e.g., using an improved statisticalvalue). In certain aspects, when information is added based oninference, the information can be weighted based on a determinedreliability rating of the user, with a notification stating theinformation appears to be correct but has not been confirmed, or acombination thereof. Actions and purposes can be also added to theentity database 234 inferentially, for example, by the processor 212monitoring user behavior after a user rejects actions suggested by theprocessor 234, and creating an action based on the user behavior (e.g.,adding an action of “find parking” when a user uses the client 110 tolocate parking near a restaurant after the user rejects selecting aproposed action of directions to the restaurant). Additionally, purposescan be grouped together automatically by the processor 212 monitoringuser behavior that indicates one or more purposes are associated with acertain entity or action. Programmatic additions include where anadministrator (e.g., a user programming the disclosed system) providesprogramming instructions to the processor 234 to detect information forentities in the entity database 234 (e.g., to handle an entity or classof entities).

The entity database 234 thus includes a listing of entities and relatedinformation (e.g., purposes and actions) that might be useful to a user,and can be accessed as a lookup table. For example, an input request tothe entity database 234 of “sushi abc” can return information thatindicates Sushi ABC is a restaurant, it is located in Mountain View, thefood it sells, its phone number, its store hours, its reviews, and pastvisits there by the user or similar users (e.g., the user's friends). Itcan further provide actions to execute, such as displaying directions toSushi ABC, reviews of Sushi ABC, the address of Sushi ABC, making areservation at Sushi ABC, finding parking at or near Sushi ABC, orsales, discounts, or promotions (including real-time promotions targetedat users of the system) at Sushi ABC.

The processor 236 of the server 130 is configured to executeinstructions, such as instructions physically coded into the processor236, instructions received from software in memory 240, or a combinationof both. With reference to FIG. 3, an example process 300 foridentifying an entity based on selected content, the processor 236 ofthe server 130 in step 310 executes instructions to receive a selectionof content from a user. A user can enter a mode to select content in theapplication 222 for interpretation by using a trigger, such as a longpress on a touchscreen input device 216 or pressing the CTRL key and amouse button on a keyboard and mouse. In certain aspects, a user canautomatically be in the content selection mode when viewing the contentfile 224 in the application 222.

For example, a user viewing a content file 224 in an application 222displayed on the output device 214 of the client 110 can select certaincontent from the content file 224, such as text, audio, or an image,using an input device 216, such as a touch screen or pointing device(e.g., a mouse). The content can be selected, for example, by the usercreating a geometric shape (e.g., circle) around the content with theuser's finger or stylus when the input device 216 is a touch input, witha cursor when the input device 216 is a mouse, or with the user's eyeswhen the input device 216 is an eye tracker. The content can also beselected, for example, based on the user's facial or physical expressionwhen the input device 216 is a camera with appropriate gesture trackingtechnology, the user's voice when the input device 216 is a microphonewith appropriate voice recognition technology, or the user's thoughtswhen the input device 216 is a brain-computer interface. The selectioncan include a portion of the content displayed from the content file 224or the entire displayed contents of the content file 224.

A copy of the selected content can then be provided to the server 130over the network 150. Contextual data regarding the selected content canalso be provided to the server 130 with the copy of the selectedcontent. Contextual data can include preceding actions of the user,including content viewing history, information on how (e.g., speed andshape) the content was selected, a current location or past location ofthe client 110, the type of the client 110, a previous use of theselected content by the user, a previous search query by the user,information on other users associated with the user, the content file224 from which the selected content was selected, the application 222 inwhich the content was selected, and the remaining content from which theselected content was selected. The previous actions (e.g., selections ofcontent, including the content itself, search queries conducted, etc.)of the user or other users as well as the previous content selected canbe stored in the entity database 234. Contextual data can also include,for example, a time of day of the selection, a current or predictedfuture weather at the location of the client 110, current news storiesor other events proximate in space and/or time to the client 110, pastactions of the client 110, predictions about future actions of the useror client 110 (e.g. based on appointments in the user's calendar, orthose of the user's known associates, or of other users in geographicalproximity to the client 110 that have elected to share their locationinformation), or other factors.

In step 330, the processor 236 of the server 130 interprets the selectedcontent (and the contextual data, if provided) to identify one or manyentities being referred to in the selected content and predict theuser's purpose(s) for selecting the entity(s). For example, the server130 can refer to the entity database 234 to interpret the selectedcontent to identify any entities in the selected content, and annotate apredict user's purpose for selecting the identified entities.

Various ways to identify a referent entity of user-selectable contentwill now be described. Selected content can include one or manyreferents, each of which is an entity. In cases where there are multipleentities identified in selected content, each entity can be weighted (or“scored”) based on a likelihood that it is a referent of the selectedcontent, with the highest weighted entity being the most likelyreferent. For example, the selected content “Miromesnil” can refer totwo entities, the Paris metro stop Miromesnil and the restaurant namedMiromesnil. The restaurant referent may be assigned a higher value(probability=0.9) than the metro stop referent (probability=0.1) becauseon contextual data indicating, among other things, that the user mayhave recently conducted a search for restaurants on the user's client110. The position of an entity in selected content can affect theentity's weighting. For example, an entity that visually appears in theapproximate center of the user's selection can be assigned a highervalue than an entity that visually appears at the periphery of theuser's selection.

The referent entity can be identified (and/or weighted) from theselected content based on the entity database 234 (e.g., a list ofentities), a current location of a client 110 on which the content isselected, a past location of the client 110, the type of the client(e.g., whether it is a mobile or non-mobile device), a previous use ofthe selected content by the user or another user, a search query,information on another user associated with the user, the file fromwhich the selected content was selected, or the remaining content (e.g.,in the content file 224) from which the selected content was selected.

In certain aspects, purposes and actions can also be assigned values forweighting in the entity database 234. For example, actions can beweighted according to their relevance to an associated entity. In suchcases, the weighting of an action for an entity can be updated (e.g.,decreased) based on an indication that a performance of the action wasabandoned or otherwise not selected by the user when the action isdisplayed to the user. Decreasing the weight of an action can be basedon a number of times an indication is received that a performance of theaction was abandoned by a user. For example, if multiple users do notselect a suggested action that is displayed an n number of times, thenthe suggested action can be reduced in weight among the other actionsassociated with the entity by a function of n.

As another example, actions can be weighted according to their relevanceto a user. For instance, if the associated value for an action can bebased on, or a function of, an estimated average amount of time it takesa user to manually perform the action (e.g., perform the action withoutthe assistance of the disclosed system). For example, if performing afirst action of making a reservation online for a restaurant would takea user five minutes, on average, and performing a second action ofcalling the restaurant would take a user 30 seconds, then the associatedvalue for the first action can be higher than the associated value forthe second action.

The estimated average amount of time to manually perform an action canbe calculated using user data. For example, a time it takes for a userto perform an action can be calculated by the processor 212 and providedto the entity database 234, which then uses the calculated time amountto determine or update the value for the associated action. The time ittakes for a user to perform an action can be also be provided manuallyto the entity database 234, such as by an administrator assigning apredetermined value.

The value can be updated periodically using recent user data. Forexample, the value can be updated on a daily basis at the end of the dayafter user data for the day has been used for calculations.

In certain aspects, the value associated with an action can be based onboth a relevance to a user and a predetermined relevance of the actionto an entity. For example, for each action, the relevance to the usercan be weighted according to one factor, and the predetermined relevanceof the action to an entity can be weighted according to another factor,and a combination of the weighted values can be used as the value forthe action. In certain aspects, other factors can also be considered.For example, the associated value for an action can be based on afrequency of times a user has selected the action, such as where a valuefor an action is increased based on a number of times the action isselected to be performed.

Returning to the identification of referent entities, an entity can alsobe identified as a referent based on the number of occurrences of anentity from the list of entities identified in the selected content. Forinstance, if an entity identifier (e.g., text) “Sushi ABC” appears inselected content more than any other identifier, then a referent of theselected content can be identified as Sushi ABC. An entity can also beidentified as a referent based on the number of occurrences of an entityfrom the list of entities identified in the remaining content. Forinstance, if an entity identifier (e.g., text) “Sushi ABC” appears inthe unselected content of the content file 224 more than any otheridentifier, then a referent of the selected content can be identified asSushi ABC.

An entity can further be identified as a referent by comparing selectedcontent of an image directly with an image or images associated withentities in the entity database 234. If the selected content of an imageappears substantially similar to an image associated with an entity inthe entity database 234 (e.g., the similarity between the selectedcontent and the image associated with the entity has a higherprobability value than the similarity between the selected content andan image associated with any other entity in the entity database 234),then the entity associated with the image can be identified as thereferent of the select content. Similarity between images can bedetermined using image recognition technology well known to those ofordinary skill in the art.

When the content file 224 is a web page, then an entity can beidentified as a referent based on other web pages that link to the webpage, other web pages that are linked to from the web page, text on theweb page, or an image on the web page. For example, a user selectscontent from a content file 224, namely, a web page that incorrectlyrefers to President George H. W. Bush as “George W. Bush.” Other webpages that link to the web page frequently and correctly include thetext “George Bush Sr.” and “George H. W. Bush.” The disclosed systemcorrectly identifies the referent entity of the selected content on theweb page, “George W. Bush,” as George H. W. Bush, even though the userselected content that incorrectly included the text “George W. Bush.”

A previous selection of an entity by another user as an appropriatereferent can also be used to identify entities as referents. The otheruser can be associated with the user that selected the content, such asby referring to the user's contact listing, the user's online socialnetwork data, or the user's electronic correspondence. For example, whena user selects content, an entity of that selected content can beidentified as a referent widget to buy if the user's friend also boughtthat widget.

An entity can further be identified as a referent from selected contentbased on whether a corresponding entity from the entity database 234 isidentified in response to a search query of the selected content. Forexample, if the selected content “Malagasy cuisine encompasses thediverse culinary traditions of the island of Madagascar; foods eaten inMadagascar reflect the influence of Southeast Asian, African, Indian,Chinese and European migrants” is entered into a search query, and thehighest ranking result from the search query is a web page titled“Malagasy cuisine,” then the referent of the selected content can beidentified as Malagasy cuisine from among the entities Malagasy cuisine,Southeast Asia, Africa, India, China, Europe, and migrants. As anotherexample, if a user selects content that is an image of a structure, anda search query of the image returns the text “Washington monument” asthe most common result, then the selected content can be identified asan image of the referent entity Washington monument.

An entity can yet further be identified as a referent from selectedcontent based on a web page previously selected in response to a searchquery that includes at least a portion of the selected content. Forexample, the content “Malagasy cuisine encompasses the diverse culinarytraditions of the island of Madagascar, foods eaten in Madagascarreflect the influence of Southeast Asian, African, Indian, Chinese andEuropean migrants” is selected by a user. The same or similar contentwas previously selected and entered into a search query by the same useror another user. In response to the search results the user selected aweb page titled “Malagasy cuisine.” The referent entity of the selectedcontent can then be identified as Malagasy cuisine.

Proximity of an entity from the entity database 234 that is identifiedin the selected content to the current location of the client 110, apast location of the client 110, or a known future location of theclient 110 (e.g., derived from a future restaurant reservation known tothe system, or an appointment in the user's calendar) can be used toidentify the entity as a referent. For example, if a user selects thecontent “Mountain View's multi-cultural restaurants, sidewalk cafes,specialty shops and professional services,” and the current location ofthe user's device is near the city of Mountain View, Calif., then thereferent entity Mountain View can be identified from among the variousentities: restaurant, café, shop, and professional services.

A previous use of the selected content by the user or another user, suchas a previous selection by the user or another user of a referent entityof content that includes the selected content, can also be used toidentify a referent entity. For example, if a user selects content thatis identical to content that has been selected in the past by anotheruser, and the other user in the past acted on a certain entityidentified from the content as the referent, then that same referententity can be presented to the current user.

An entity can further be identified as a referent based on a proximityof entities (e.g., from the entity database 234), to the currentlocation of the client 110, that are identified in the selected contentthat have at least a predetermined ranking value in a result listing ofa search query that includes at least a portion of the selected content.For example, if a user selects the content “Mountain View'smulti-cultural restaurants, sidewalk cafes, specialty shops andprofessional services,” the current location of the user's device isnear the city of Mountain View, Calif., and a search of the selectedcontent returns “Mountain View” as the top ranking result, then thereferent Mountain View can be identified from among the variousentities: restaurant, café, shop, and professional services.

If an entity is not appropriately identified as a referent in responseto a selection of content by a user, then the user can be provided withan interface for manually identifying the entity intended to be thereferent. For example, the interface can be a text box. If the referententity is not present in the entity database 234, then the entity can beadded to the entity database 234, for example, manually by the user. Theuser can also be provided with an opportunity to correct or otherwiseannotate the selected content to disambiguate the selected content orprovide additional information that would facilitate the identificationof a referent entity. For example, the user can de-select a portion ofthe selected content, select additional content, or manually provideinput (e.g., in a displayed text box) that provides additionalidentifying information for the user's intended referent.

Returning to FIG. 3, and continuing as part of step 330, an identifierof the identified referent entity(s) is provided to the client 110 todisplay to the user. The user then selects the appropriate entity fromthe displayed referent entity(s). The user-selected entity is thenprovided to the server 130 in step 350 along with information on acontext in which the entity was selected. Next, in step 350 theprocessor 236 is configured to execute instructions to provide, based onthe context in which the entity was selected, actions associated withthe predicted user purpose(s) for the user-selected entity to present tothe user via the client 110, so that the client 110 in step 370 canexecute any such presented action selected by the user. The context caninclude the context a current location of the client 110, a pastlocation of the client 110, the type of the client 110, a previousaction associated with the entity taken by the user or another user, asearch query, information on another user associated with the user, thefile from which the user-selectable content was selected, and theremaining content from which the user-selectable content was selected.In certain aspects, after one or many entities are selected by the useras the appropriate referent(s), the predicted purpose associated witheach entity can be identified (e.g., by referring to the entity database234) and included with the context information. In cases where there aremultiple purposes identified for an entity, each purpose can be weightedbased on a likelihood that it is the purpose for selecting the entity,with the highest weighted purpose being the most likely purpose. Forexample, returning to the example of the restaurant Miromesnil, apurpose of making a reservation may be assigned a higher value(probability=0.75) than viewing the restaurant history(probability=0.25) because the user's client 110 is determined to begeographically close to the restaurant (e.g., in Paris).

The action(s) associated with each predicted purpose can then beprovided for display to the user on the client 110 for later selectionand execution by the user. The action to be executed is associated withthe entity and selected based on the context in which the entity wasselected on the client 110 as the referent. Specifically, the processor236 of the server 130 is configured to identify the action based on acontext in which the entity is selected by a user from the list ofsuggested entities. The processor 236 then provides, for display in step370, an identifier of the action to the client 110 for selection by theuser. In certain aspects, the identifiers of the actions associated withan entity can be provided along with the identifier of the entity as areferent in step 330 described above. In certain aspects, the processor236 in step 370 provides, for display, an identifier of the action tothe client 110, wherein the action was already selected by the server130.

When the entity is a product, such as a bicycle, the actions that can bepresented to the user on the client 110 can include providing a reviewof the product, identifying a seller of the product, providing a pricefor the product, or providing an offer (e.g., discount or coupon)associated with the product. When the product is software, theassociated actions can also include providing a location for downloadingthe software, or installing the software. When the entity is a service,such as watching a movie or a plumber for hire, the actions that can bepresented to the user on the client 110 include providing a review ofthe service, identifying an availability of the service (e.g., showtimes), identifying a location where the service is being provided(e.g., an address of the plumber), or providing an option to purchasethe service (e.g., purchasing tickets for the movie or rates offered bythe plumber). When the entity is a location, such as a restaurant, theactions that can be presented to the user on the client 110 includeproviding a review of the location, identifying an address of thelocation, providing directions to the location, providing information onavailability of the location (e.g., making a reservation), providinginformation on parking associated with the location, or providing anoffer associated with the location. When the entity is a person, theactions that can be presented to the user on the client 110 includeproviding contact information for the person, a description of theperson, an image of the person, or information on a relationship withthe person (e.g., in an online social network).

FIG. 3 sets forth an example process 300 for identifying an entity basedon selected content using the example client 110 and server 130 of FIG.2. An example will now be described using the example process 300 ofFIG. 3, a client 110 that is a tablet computer with a touchscreeninterface, an application 222 that is a web browser, and a content file224 that is a web page for a restaurant Sushi ABC.

The process 300 proceeds to step 310 when a user on a tablet computer110 opens a web browser 222 to view a web page 224. As illustrated inFIG. 4A, an example screenshot 400 of the web page 224, the user selectscontent 406 by circling the content 406 with the user's finger using atouch interface 216 of the tablet computer 110. The tablet computer 110then provides visual feedback by overlaying a line 404 where the userdraws the circle. In certain aspects not illustrated, the line 404 canbe made more bold, less bold, narrower, wider, etc., to indicateuncertainty about the user's intended line, and/or the selected contentcan also be highlighted by emboldening, enlarging, italicizing, oremphasizing the selected content in another way. A copy of selectedcontent (e.g., the text “Sushi ABC offers over 120 savory satisfactionbetter would rather skip the easy Sushi ABC guarantees the a lower pricethan the order right before your only each”) is provided to the server130, and in step 330 the selected content from the web page 224 isinterpreted to identify any entity(s) and predicted or expressed (e.g.,via user selection) user purpose(s).

Specifically, a search query is run of the selected content, and thehighest value result is a web page for the restaurant Sushi ABC. Theuniform resource locator (URL) of the web page for the restaurant SushiABC is then evaluated in the entity database 234, and the entitydatabase 234 returns the entity Sushi ABC as the entity being referredto due to the Sushi ABC entry in the entity database 85 including theidentical URL for the restaurant. The entity Sushi ABC is associatedwith a descriptors “location” and “restaurant” in the database. Theentity Sushi ABC is also associated in the entity database 234 with thepurposes of going to the restaurant, calling the restaurant, savinginformation on the restaurant, and offers associated with therestaurant. These purposes are associated with the actions of providingdirections to the restaurant, dialing the restaurant, adding therestaurant information to a contacts database, and displaying offersassociated with the restaurant, respectively. An identifier of theentity Sushi ABC, namely the text “Restaurant Sushi ABC,” is thenprovided by the server 130 to the tablet computer 110 for display, asillustrated in FIG. 4B, an example screenshot 410 of a listing ofreferent entities in a referent menu on the tablet computer 110. Thereferent menu includes an option for a user to select the referentidentifier “Restaurant Sushi ABC” 414 by pressing “Go!,” or select tocopy 412 the selected content.

The user selects the entity Sushi ABC from the listing by pressing onthe text “Restaurant Sushi ABC” 414. In this example, the list includesone referent entity. The user-selected entity Sushi ABC is then providedto the server 130 along with information on the context (e.g., locationof the tablet computer 110) in which the selection was made on thetablet computer 110, and in response, an action selection interface thatincludes a listing of actions 422 associated with the entity Sushi ABCis provided by the server 130 to the client 110 for display. The actionsare displayed in order to be performed with/for the entity Sushi ABC, asillustrated in the example screenshot 420 of FIG. 4C. The actionsinclude obtaining directions to the restaurant by pressing “Go!” 424,calling the restaurant from the tablet computer 110 by pressing “Call!”426, adding information on the restaurant to the user's contactsdatabase on the tablet computer 110 by pressing “Add to Contacts” 428,and viewing a special offer for the restaurant by pressing “SpecialOffer for dinner TONIGHT!” 430. In step 370, the user presses “Go!” 424and, as illustrated in the example screenshot 440 of FIG. 4D, the tabletcomputer 110 displays driving directions from the user's currentlocation 442 to the address of the restaurant 444.

In certain aspects, the disclosed interface can provide suggestedactions for display according to their value to the user. As discussedabove, each suggested action can have an associated value. Theassociated value can be considered when the listing of actions 422 isprovided for display to a user. Specifically, the processor 236 of theserver 130 or the processor 212 of the tablet computer 110 is configuredto execute instructions, such as instructions physically coded into theprocessor 236/212, instructions received from software in memory240/222, or a combination of both, to receive content selected by a userfrom a device, identifying actions to be executed that are associatedwith an entity identified as a referent from the selected content,wherein each action has an associated value to the user, and provide,for display, an identifier for each action in at least a subset of theactions based on their associated values. The illustration 450 of FIG.4E illustrates a listing of actions 422 in text format where thesuggested actions 452, 428, 424, 426, 454, and 430 are ordered in thedisplay based on their associated values. The values take intoconsideration the relevance of the action to the user (e.g., based onhow many times a user has selected the action) and the average time itwould take a user to manually perform the action. The action “Make aReservation” 452 is determined to have the highest value because it isboth commonly selected by users and requires the greatest amount of timeto perform manually amongst the other actions displayed. Similarly, theaction “Add to Contacts” 428 is determined to have the second highestvalue because it is also commonly selected by users and requires asignificant amount of time to perform manually, but not as much time asthe first action 452. The next two actions, “Go!” 424 and “Call” 426,have the next highest values because they are very frequently selectedby users, and even though they would not take a significant amount oftime to perform manually. The final two actions, “View Menu” 454 and“Special Offer for dinner TONIGHT!” 430 are displayed in the last twopositions, respectively, because they either do not take a significantamount of time to perform and/or they are not frequently selected byusers. In certain aspects, actions having an associated value that arebelow a predetermined threshold are not provided for display.

FIG. 5 is a block diagram illustrating an example computer system 500with which the client 110 and server 130 of FIG. 2 can be implemented.In certain aspects, the computer system 500 may be implemented usinghardware or a combination of software and hardware, either in adedicated server, or integrated into another entity, or distributedacross multiple entities.

Computer system 500 (e.g., client 110 and server 130) includes a bus 508or other communication mechanism for communicating information, and aprocessor 502 (e.g., processor 212 and 236) coupled with bus 508 forprocessing information. By way of example, the computer system 500 maybe implemented with one or more processors 502. Processor 502 may be ageneral-purpose microprocessor, a microcontroller, a Digital SignalProcessor (DSP), an Application Specific Integrated Circuit (ASIC), aField Programmable Gate Array (FPGA), a Programmable Logic Device (PLD),a controller, a state machine, gated logic, discrete hardwarecomponents, or any other suitable entity that can perform calculationsor other manipulations of information.

Computer system 500 can include, in addition to hardware, code thatcreates an execution environment for the computer program in question,e.g., code that constitutes processor firmware, a protocol stack, adatabase management system, an operating system, or a combination of oneor more of them stored in an included memory 504 (e.g., memory 220 and232), such as a Random Access Memory (RAM), a flash memory, a Read OnlyMemory (ROM), a Programmable Read-Only Memory (PROM), an Erasable PROM(EPROM), registers, a hard disk, a removable disk, a CD-ROM, a DVD, orany other suitable storage device, coupled to bus 508 for storinginformation and instructions to be executed by processor 502. Theprocessor 502 and the memory 504 can be supplemented by, or incorporatedin, special purpose logic circuitry.

The instructions may be stored in the memory 504 and implemented in oneor more computer program products, i.e., one or more modules of computerprogram instructions encoded on a computer readable medium for executionby, or to control the operation of, the computer system 500, andaccording to any method well known to those of skill in the art,including, but not limited to, computer languages such as data-orientedlanguages (e.g., SQL, dBase), system languages (e.g., C, Objective-C,C++, Assembly), architectural languages (e.g., Java, .NET), andapplication languages (e.g., PHP, Ruby, Perl, Python). Instructions mayalso be implemented in computer languages such as array languages,aspect-oriented languages, assembly languages, authoring languages,command line interface languages, compiled languages, concurrentlanguages, curly-bracket languages, dataflow languages, data-structuredlanguages, declarative languages, esoteric languages, extensionlanguages, fourth-generation languages, functional languages,interactive mode languages, interpreted languages, iterative languages,list-based languages, little languages, logic-based languages, machinelanguages, macro languages, metaprogramming languages, multiparadigmlanguages, numerical analysis, non-English-based languages,object-oriented class-based languages, object-oriented prototype-basedlanguages, off-side rule languages, procedural languages, reflectivelanguages, rule-based languages, scripting languages, stack-basedlanguages, synchronous languages, syntax handling languages, visuallanguages, wirth languages, embeddable languages, and xml-basedlanguages. Memory 504 may also be used for storing temporary variable orother intermediate information during execution of instructions to beexecuted by processor 502.

A computer program as discussed herein does not necessarily correspondto a file in a file system. A program can be stored in a portion of afile that holds other programs or data (e.g., one or more scripts storedin a markup language document), in a single file dedicated to theprogram in question, or in multiple coordinated files (e.g., files thatstore one or more modules, subprograms, or portions of code). A computerprogram can be deployed to be executed on one computer or on multiplecomputers that are located at one site or distributed across multiplesites and interconnected by a communication network. The processes andlogic flows described in this specification can be performed by one ormore programmable processors executing one or more computer programs toperform functions by operating on input data and generating output.

Computer system 500 further includes a data storage device 506 such as amagnetic disk or optical disk, coupled to bus 508 for storinginformation and instructions. Computer system 500 may be coupled viainput/output module 510 to various devices. The input/output module 510can be any input/output module. Example input/output modules 510 includedata ports such as USB ports. The input/output module 510 is configuredto connect to a communications module 512. Example communicationsmodules 512 (e.g., communications module 218 and 238) include networkinginterface cards, such as Ethernet cards and modems. In certain aspects,the input/output module 510 is configured to connect to a plurality ofdevices, such as an input device 514 (e.g., input device 216) and/or anoutput device 516 (e.g., output device 214). Example input devices 514include a keyboard and a pointing device, e.g., a mouse or a trackball,by which a user can provide input to the computer system 500. Otherkinds of input devices 514 can be used to provide for interaction with auser as well, such as a tactile input device, visual input device, audioinput device, or brain-computer interface device. For example, feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, tactile,or brain wave input. Example output devices 516 include display devices,such as a CRT (cathode ray tube) or LCD (liquid crystal display)monitor, for displaying information to the user.

According to one aspect of the present disclosure, the client 110 andserver 130 can be implemented using a computer system 500 in response toprocessor 502 executing one or more sequences of one or moreinstructions contained in memory 504. Such instructions may be read intomemory 504 from another machine-readable medium, such as data storagedevice 506. Execution of the sequences of instructions contained in mainmemory 504 causes processor 502 to perform the process steps describedherein. One or more processors in a multi-processing arrangement mayalso be employed to execute the sequences of instructions contained inmemory 504. In alternative aspects, hard-wired circuitry may be used inplace of or in combination with software instructions to implementvarious aspects of the present disclosure. Thus, aspects of the presentdisclosure are not limited to any specific combination of hardwarecircuitry and software.

Various aspects of the subject matter described in this specificationcan be implemented in a computing system that includes a back endcomponent, e.g., as a data server, or that includes a middlewarecomponent, e.g., an application server, or that includes a front endcomponent, e.g., a client computer having a graphical user interface ora Web browser through which a user can interact with an implementationof the subject matter described in this specification, or anycombination of one or more such back end, middleware, or front endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, e.g., a communicationnetwork. The communication network (e.g., communication network 150) caninclude, for example, any one or more of a personal area network (PAN),a local area network (LAN), a campus area network (CAN), a metropolitanarea network (MAN), a wide area network (WAN), a broadband network(BBN), the Internet, and the like. Further, the communication networkcan include, but is not limited to, for example, any one or more of thefollowing network topologies, including a bus network, a star network, aring network, a mesh network, a star-bus network, tree or hierarchicalnetwork, or the like. The communications modules can be, for example,modems or Ethernet cards.

Computing system 500 can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.Computer system 500 can be, for example, and without limitation, adesktop computer, laptop computer, or tablet computer. Computer system500 can also be embedded in another device, for example, and withoutlimitation, a mobile telephone, a personal digital assistant (PDA), amobile audio player, a GPS receiver, a video game console, and/or atelevision set top box.

The term “machine-readable storage medium” or “computer readable medium”as used herein refers to any medium or media that participates inproviding instructions to processor 502 for execution. Such a medium maytake many forms, including, but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media include, forexample, optical or magnetic disks, such as data storage device 506.Volatile media include dynamic memory, such as memory 504. Transmissionmedia include coaxial cables, copper wire, and fiber optics, includingthe wires that comprise bus 508. Common forms of machine-readable mediainclude, for example, floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD, any other opticalmedium, punch cards, paper tape, any other physical medium with patternsof holes, a RAM, a PROM, an EPROM, a FLASH EPROM, any other memory chipor cartridge, or any other medium from which a computer can read. Themachine-readable storage medium can be a machine-readable storagedevice, a machine-readable storage substrate, a memory device, acomposition of matter effecting a machine-readable propagated signal, ora combination of one or more of them.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of what may be claimed, but ratheras descriptions of particular implementations of the subject matter.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the aspects described above should not be understood asrequiring such separation in all aspects, and it should be understoodthat the described program components and systems can generally beintegrated together in a single software product or packaged intomultiple software products.

The subject matter of this specification has been described in terms ofparticular aspects, but other aspects can be implemented and are withinthe scope of the following claims. For example, the actions recited inthe claims can be performed in a different order and still achievedesirable results. As one example, the processes depicted in theaccompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. In certainimplementations, multitasking and parallel processing may beadvantageous. Other variations are within the scope of the followingclaims.

These and other implementations are within the scope of the followingclaims.

What is claimed is:
 1. A computer-implemented method for proposingactions to a user to select based on the value of the actions to theuser, the method comprising: receiving content selected by a user from adevice; identifying a plurality of actions to be executed that areassociated with a referent entity identified from the selected content,wherein each action has an associated value to the user and wherein theassociated value for at least two of the plurality of actions is afunction of an estimated average amount of time it takes a user tomanually perform each of the at least two actions; and providing, fordisplay, an identifier of each of at least a subset of the actions basedon their associated values.
 2. The computer-implemented method of claim1, wherein an identifier for an action having an associated value belowa predetermined threshold is not provided for display.
 3. Thecomputer-implemented method of claim 1, wherein the identifier for eachaction comprises text provided for display in a list format.
 4. Thecomputer-implemented method of claim 3, wherein the identifiers for eachaction in the list are displayed in decreasing order of their associatedvalues.
 5. The computer-implemented method of claim 1, wherein theestimated average amount of time is calculated using user data.
 6. Thecomputer-implemented method of claim 5, wherein the estimated averageamount of time is periodically updated using recent user data.
 7. Thecomputer-implemented method of claim 1, wherein the estimated averageamount of time for at least one action is a predetermined value.
 8. Thecomputer-implemented method of claim 1, wherein each of the plurality ofactions is further weighted based on a predetermined relevance of theaction to the referent entity.
 9. The computer-implemented method ofclaim 1, wherein the associated value for at least one action is basedon a frequency of times a user has selected the action.
 10. A system forproposing actions to a user to select based on the value of the actionsto the user, the system comprising: a memory comprising user-selectablecontent; and a processor configured to: receive content selected by auser from a device; identify a plurality of actions to be executed thatare associated with a referent entity identified from the selectedcontent, wherein each action has an associated value to the user andwherein the associated value for at least two of the plurality ofactions is a function of an estimated average amount of time it takes auser to manually perform each of the at least two actions; and provide,for display, an identifier of each of at least a subset of the actionsbased on their associated values.
 11. The system of claim 10, wherein anidentifier for an action having an associated value below apredetermined threshold is not provided for display.
 12. The system ofclaim 10, wherein the identifier for each action comprises text providedfor display in a list format.
 13. The system of claim 12, wherein theprocessor is configured to provide, for display, identifiers for eachaction in the list in decreasing order of their associated values. 14.The system of claim 10, wherein the processor is configured to calculatethe estimated average amount of time using user data.
 15. The system ofclaim 14, wherein the processor is configured to periodically update theestimated average amount of time using recent user data.
 16. The systemof claim 10, wherein the estimated average amount of time for at leastone action is a predetermined value.
 17. The system of claim 10, whereineach of the plurality of actions is further weighted based on apredetermined relevance of the action to the referent entity.
 18. Thesystem of claim 10, wherein the associated value for at least one actionis based on a frequency of times a user has selected the action.
 19. Anon-transitory machine-readable storage medium comprisingmachine-readable instructions for causing a processor to execute amethod for proposing actions to a user to select based on the value ofthe actions to the user, the method comprising: receiving contentselected by a user from a device; identifying a plurality of actions tobe executed that are associated with a referent entity identified fromthe selected content, wherein each action has an associated value to theuser that is a function of an estimated average amount of time it takesa user to manually perform the action; and providing, for display, in alist format at least one of a text or image that identifies each of atleast a subset of the actions, wherein the text or image for each actionin the list is displayed in decreasing order of the associated value forthe action.